Multi-mode video event indexing

ABSTRACT

Multi-mode video event indexing includes determining a quality of object distinctiveness with respect to images from a video stream input. A high-quality analytic mode is selected from multiple modes and applied to video input images via a hardware device to determine object activity within the video input images if the determined level of detected quality of object distinctiveness meets a threshold level of quality, else a low-quality analytic mode is selected and applied to the video input images via a hardware device to determine object activity within the video input images, wherein the low-quality analytic mode is different from the high-quality analytic mode.

BACKGROUND

The present invention relates to the analysis of activities in videos,and more particularly to accurately determining and distinguishingobject movements and activities represented thereby.

Video surveillance enables object monitoring through video displays ofone or more areas remote from a human monitor. Exemplary applicationsinclude security surveillance of public and private areas, for exampleparking lots for human and vehicle movements, assembly areas such astrain stations and entertainment halls for abandoned baggage or objects,borders and doorways for unauthorized entry, secured areas forunauthorized vehicle or object movements and removals, etc. However,human review and analysis of video feeds is time consuming and perhapsinefficient with respect to human resources allocations, and accordinglyit is desirable to implement automated systems for video analysis.

Automated analysis of videos for determining object movements,activities and behaviors presents a number of challenges. Variablevolumes of activity data, weather conditions, human or object crowdingwithin a scene, geographical area features and other factors often proveproblematic for accurate results in making such determinations throughvideo analytics algorithms.

BRIEF SUMMARY

One embodiment of a method for multi-mode video event indexing includesdetermining a quality of object distinctiveness with respect to imagesfrom a video stream input. A high-quality analytic mode is selected frommultiple modes and applied to video input images via a hardware deviceto determine object activity within the video input images if thedetermined level of detected quality of object distinctiveness meets athreshold level of quality, else a low-quality analytic mode is selectedand applied to the video input images via a hardware device to determineobject activity within the video input images, wherein the low-qualityanalytic mode is different from the high-quality analytic mode.

In another embodiment, a computer system for multi-mode video eventindexing comprises a processing unit, computer readable memory and acomputer readable storage system having program instructions todetermine a quality of object distinctiveness with respect to imagesfrom a video stream input; select from a plurality of video analyticsmodes and apply a high-quality analytic mode to the video input imagesto determine object activity within the video input images if thedetermined level of detected quality of object distinctiveness meets athreshold level of quality. Program instructions are also to select alow-quality analytic mode from the plurality of video analytics modesand apply the low-quality analytic mode to the video input images todetermine object activity within the video input images if thedetermined level of detected quality of object distinctiveness does notmeet the threshold level of quality, the low-quality analytic modedifferent from the high-quality analytic mode.

In another embodiment, a computer program product for multi-mode videoevent indexing comprises a computer readable storage medium and programinstructions stored thereon to determine a quality of objectdistinctiveness with respect to images from a video stream input; selecta high-quality analytic mode from a plurality of video analytics modesand apply the high-quality analytic mode to the video input images todetermine object activity within the video input images if thedetermined level of detected quality of object distinctiveness meets athreshold level of quality; and select a low-quality analytic mode fromthe plurality of video analytics modes and apply the low-qualityanalytic mode to the video input images to determine object activitywithin the video input images if the determined level of detectedquality of object distinctiveness does not meet the threshold level ofquality, the low-quality analytic mode different from the high-qualityanalytic mode.

In another embodiment, a service for multi-mode video event indexingprovides a computer infrastructure that determines a quality of objectdistinctiveness with respect to images from a video stream input;selects a high-quality analytic mode from a plurality of video analyticsmodes and applies the high-quality analytic mode to the video inputimages to determine object activity within the video input images if thedetermined level of detected quality of object distinctiveness meets athreshold level of quality; and selects a low-quality analytic mode fromthe plurality of video analytics modes and applies the low-qualityanalytic mode to the video input images to determine object activitywithin the video input images if the determined level of detectedquality of object distinctiveness does not meet the threshold level ofquality, wherein the low-quality analytic mode is different from thehigh-quality analytic mode.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

These and other features of this invention will be more readilyunderstood from the following detailed description of the variousaspects of the invention taken in conjunction with the accompanyingdrawings in which:

FIG. 1 illustrates a method or system for determining object movementsaccording to the present invention.

FIG. 2 illustrates an embodiment according to the present invention.

FIG. 3 illustrates another embodiment according to the presentinvention.

FIGS. 4 a and 4 b are graphical illustrations of bounding box distancemeasures according to the present invention.

FIG. 5 is a graphical illustration of a rule-based object classificationaccording to the present invention.

FIG. 6 is an illustration of an embodiment of a tripwire classifieraccording to the present invention.

FIG. 7 is a computerized implementation of an embodiment of the presentinvention.

The drawings are not necessarily to scale. The drawings are merelyschematic representations, not intended to portray specific parametersof the invention. The drawings are intended to depict only typicalembodiments of the invention, and therefore should not be considered aslimiting the scope of the invention. In the drawings, like numberingrepresents like elements.

DETAILED DESCRIPTION

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, in abaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including, but not limited to, wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the present invention are described below with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

Historically, compliance of object activity with policies, regulations,etc. has typically been enforced through direct human surveillance. Forexample, safety and security personnel may watch cameras trained oncertain areas to discover deviations from safety policies, trespassing,theft, unauthorized access to restricted areas, etc. However, humanvisual attention may be ineffective, particularly for large volumes ofvideo data. Due to many factors, illustratively including an infrequencyof activities of interest, a fundamental tedium associated with the taskand poor reliability in object tracking in environments with visualclutter and other distractions, human video surveillance may be bothexpensive and ineffective.

Automated video surveillance systems and methods are also proposed orknown, wherein computers or other programmable devices directly analyzevideo data and attempt to determine the occurrence of activities ofconcern. However, determining and differentiating human and objectmovements within a video stream object in prior art automated videosurveillance systems and methods is often not reliable in realistic,real-world environments and applications, sometimes due to clutter, pooror variable lighting and object resolutions and distracting competingvisual information.

Referring now to FIG. 1, a dual-mode embodiment of a multi-mode videoevent indexing method, infrastructure or system for determination ofobject movements according to the present invention is illustrated. At10 a quality of object distinctiveness is determined with respect toimages from a video stream input, and accordingly an appropriate mode ofvideo analytics is selected for application to the video stream input.The present example makes a binary high or low quality determination,wherein a high quality-based analytic mode 12 is selected fordetermining movement of objects, and otherwise a low-quality-based mode16 is selected. It will be understood that detecting an image qualityrelative to object distinctiveness within an image comprehends andselecting an appropriate mode of video analytics according to thepresent invention comprehends more than two different modes or levels;for example, three different modes may be selectable for each of low,medium and high levels or qualities of object distinctiveness, or lowand high quality modes may be applied in combination for a medium level,and still more options may be presented.

Embodiments of the present invention also provide for different alertgenerations for the different modes: in the present example,implementation of the object tracking-based mode 12 results ingenerating alerts at 14 as a function of tracking-based analytics,whereas implementation of the non-tracking-based mode 16 results ingenerating alerts at 18 as a function of the non-tracking-basedanalytics. Analytic results at 12 and 16 and alert generation results at14 and 18 may be written to an index at 19 for reporting and furtheranalysis, for example to a database for data analysis.

The present invention provides for the automatic selection of anappropriate video analytic process in response to a quality of objectdistinctiveness indicating reliability in results, including of alertsand alarm generation, in identifying, recognizing and tracking objectmovements. The determination of one or more qualities of objectdistinctiveness with respect to the video images may comprehend one ormore of a variety of criteria and processes. For example, FIG. 2illustrates an embodiment according to the present invention forautomatically switching to an appropriate video parsing device at 20 inresponse to determining and distinguishing different weather conditionsof the video image, including but not limited to sunny, cloudy orchanging levels of cloudiness (e.g. fully cloudy versus partiallycloudy), rainy, snowy, etc. Recognizing a specified weather condition at20 may be accomplished through video analytics applied to images of thevideo input, for example determining a sunny or well-lit scene throughmeasuring a light intensity of the video image and comparing it to athreshold illumination value. The recognition at 20 may also beresponsive to other inputs or selection processes, for example aseparate weather reporting application may report clear skies duringcertain hours known to comprise appropriate daylight hours of the scene.

When the weather is sunny, outdoor moving objects often possess castedshadows with clear boundaries, and thus recognition of sunny daylightconditions at 20 results in selection of a strong-shadow analytic modeor device 22 that provides for object detection and tracking, one thathandles strong shadows well, for example by removing or otherwiseaccounting for strong shadows in image analytics, and wherein alertdetermination at 24 is responsive to the strong-shadow analytic device22 outputs. Otherwise, if the scene is not sunny, for example cloudy,rainy, snowy, foggy, twilight, dawn, etc., where the lighting is dimmerand object shadows are not as distinctive, then a low-light analyticmode or device 26 is selected, for example one that parses video eventswithout using shadow removal techniques, and wherein alert determinationat 28 is responsive to the low-light analytic device 26 outputs.Analytic results at 22 and 26 and alert generation results at 24 and 28are accordingly written to an index at 30 for reporting and furtheranalysis.

Embodiments of the present invention may determine a level of objectactivity within a video image through background subtraction methods,and then select an appropriate mode of object movement determination;for low levels of activity, each moving object may be tracked as itmoves across a scene, but to handle high-activity scenes (for example,those typical in certain hours of the day in urban scenarios), theembodiments determine object movements through object appearanceattribute retrieval and processing without object tracking. Thus, FIG. 3illustrates an embodiment which applies background subtraction (BGS) toa video stream input at 102 in order to detect foreground objectactivity as a difference between current frame image information andstatic information of a contextual background of the image. At 104 alevel of foreground object activity is determined after the backgroundsubtraction, and accordingly an appropriate mode of video analytics isselected for application to the video. Thus, an object tracking-basedmode Color Field Tracker 124 is selected for lower levels of activityappropriate for determining movement of objects through trackingdistinct foreground objects; and a non-tracking-based mode Color BGSClassifier 132 is selected for higher activity levels, one moreappropriate for determining object movements in lower quality fromextracted appearance attributes of foreground objects, without objecttracking.

Embodiments as illustrated in FIG. 3 may also be appropriate forapplication in distinguishing weather or illumination scene conditions,for example as described with respect to the embodiment of FIG. 2,wherein rather than select between strong shadow modes/devices 22 andlow-light modes/devices 26, embodiments may select between trackingmodes/devices 124 and non-tracking modes/devices 132. Thus, recognitionat 104 that video images are acquired under rain, snow or fog conditionsmay indicate use of high-level, non-tracking modes/devices, as opposedto sunny and clear weather scene images, more suitable for use of thelow-level, tracking mode/devices.

Determining a quality of image of the video input at 104 may comprisedetermining an amount of illumination in a scene of the video input andselecting an appropriate mode by comparison to a specified level ofillumination; for example, nighttime or more poorly illuminated scenesmay result in higher foreground-to-background ratios relative to sunlitor better illuminated scenes, indicating use of the high-level,non-tracking mode 132. Video inputs from moving video cameras may alsoindicate use of high-level, non-tracking modes/devices 132, as opposedto those taken from stationary or relatively more stable cameras moresuitable for use of the low-level, tracking modes/devices 124.

Other image qualities may also indicate relative object activity at 104.For example, higher densities of foreground objects or activity mayresult from cluttered or poor image quality, triggering a “high-level”choice of non-tracking analytics 124, even where low levels of objectmotion may be occurring.

Some embodiments of the present invention perform activity leveldetection at 104 through use of a switch determination module that takesthe results of the BGS at 102 and looks at density of the image todetermine an activity level or quality as a function of a ratio offoreground to background. Thus, some examples may utilize a thresholdforeground density value (e.g. a percentage of 60%) relative to theentire image, densities above which triggering selection of thehigh-level mode, non tracking analytics at 104. Another example uses 40%as a threshold density percentage, and still other percentages may bepracticed.

Embodiments of the present invention may also perform activity leveldetection at 104 as a function of a stability of ratio data, for examplethrough use of a temporal consistency analysis. Thus, some embodimentsmay require that one or more threshold density percentages (60%, 40%,etc.) be exceeded at 10 at least over one or more specified time periodsand, if not, the low level/tracking-based analytics may still beselected even if the threshold has been exceeded. In one example, if athreshold density (e.g. 60%) met at a first, earlier time drops to aspecified minimum value (e.g. to no more than 30%) at a subsequent,second time within an elapsed time period chosen to represent astability period or duration (e.g. within one or two seconds or othertime periods), then the low level/tracking-based analytics may also beselected at 10 even though the threshold density was exceeded.Stabilities and/or densities may also be considered as averages overtime, and in some embodiments by further weighting some time periodsrelative to others (for example, weighting more recent times or averagesof times relatively higher than older times and/or averages).

Activities and levels thereof may comprehend a variety ofcharacteristics and actions, for example numbers of object moving, speedor acceleration of one or more objects, relative to static background orother moving objects, relative quality of image (e.g. different relativelighting levels, such as from sunshine, clouds, nighttime, fog;occlusions from rain, snow or other environmental elements and factors,etc.). Other examples of or causes for divergentforeground-to-background ratio values may also occur, and the presentinvention is adaptable to respond to said causal agents by selecting theappropriate high, low or even intermediate mode.

More particularly, embodiments of the present invention may determine alevel of object activity within a video image through backgroundsubtraction methods, and then select an appropriate mode of objectmovement determination; for low levels of activity, each moving objectmay be tracked as it moves across a scene, but to handle high-activityscenes (for example those typical in certain hours of the day in urbanscenarios), the embodiments determine object movements through objectappearance attribute retrieval and processing without object tracking.

The background subtractor 102 may apply a statistical approach tobackground subtraction, for example as taught by T. Horprasert, D.Harwood and L. S. Davis in “A statistical approach for real-time robustbackground subtraction and shadow detection,” ICCV Frame-Rate Workshop,1999. Such embodiments make background subtraction robust toillumination changes by statistically modeling the backgroundindependently at each pixel. An exemplary estimation process thuscomputes the brightness distortion and color distortion inred-green-blue (RGB) color space wherein each pixel {i} is modeled by a4-tuple (E_(i), s_(i), a_(i), b_(i)), where {E_(i)} is a vector with themeans of the pixel's red, green, and blue components computed over anumber {N} of background frames; {s_(i)} is a vector with the standarddeviations of the color values; {a_(i)} is the variation of thebrightness distortion; and {b_(i)} is the variation of the chromaticitydistortion.

By comparing the difference between the background image and the currentimage, a given pixel {i} may be classified into one of four categories:original background, shaded background or shadow, highlightedbackground, and foreground. Categorization thresholds may be calculatedautomatically by statistical learning wherein histograms of a normalizedbrightness distortion, a normalized chromaticity distortion areconstructed from combined data through a long sequence captured during abackground learning period, and wherein thresholds are automaticallyselected according to the desired detection rate as a function of thehistograms. Foreground pixels may thus be passed to the appropriateanalytics mode chosen at 104, with remaining pixels grouped together asbackground, and isolated pixels may be removed and a morphologicalclosing operator applied to join nearby foreground pixels.

Active background estimation may also be provided to deal with objectsmoving in training images wherein a first frame is stored as a prototypebackground image and differenced with subsequent training frames, theareas of significant difference defining moving objects excluded whenthe statistical background model is constructed. Variations inillumination not seen in the training set are handled by modifying thebackground subtraction method and process algorithms by an overall gaincontrol that applies a global scaling factor to pixel intensities beforecomparing them to the stored means, the scale factor calculated onnon-foreground regions of a previous image under an assumption thatlighting changes between adjacent frames are small; and further throughbackground adaptation employed by blending in pixel values of currentnon-foreground regions, in one aspect slowly learning local changes inappearance not attributable to moving objects. Said processes reduce thesensitivity of the background estimation algorithms to lighting changesbetween and during datasets.

Embodiments of the background subtractor 102 may also apply a Gaussianmixture model approach with respect to each pixel in a video imagescene. In one example, for a mixture of {K} Gaussians chosen from 3 to5, the probability of a pixel {X} at time {t}, may be described as:

$\begin{matrix}{{{P\left( X_{t} \right)} = {\sum\limits_{i = 1}^{K}{\omega_{i,t}*{\eta\left( {X_{t},\mu_{i,t},\sum_{i,t}} \right)}}}},{where}} & (1) \\{{{\eta\left( {X_{t},\mu,\sum} \right)} = {\frac{1}{\left( {2\pi} \right)^{\frac{n}{2}}{\sum }^{\frac{1}{2}}}{\mathbb{e}}^{{- \frac{1}{2}}{({X_{t} - \mu_{t}})}^{T}{\sum^{- 1}{({X_{t} - \mu_{t}})}}}}},{and}} & (2) \\{\omega_{i,t} = {{\left( {1 - \alpha} \right)\omega_{i,{t - 1}}} + {{\alpha\left( M_{k,t} \right)}.}}} & (3)\end{matrix}$

Wherein {μ} is the mean, {α} is the learning rate and {M_(k,t)} is 1 forthe model which matched, and 0 for the remaining models. By assuming thered, green, and blue pixel values are independent and have the samevariances:Σ_(k,t)=Σ_(k) ²I.  (4)

After the Gaussians are ordered by the value of {ω/α}, the first {B}distributions are chosen as the background model, where

$\begin{matrix}{{B = {{argmin}_{b}\left( {{\sum\limits_{k = 1}^{b}\omega_{k}} > T} \right)}},} & (5)\end{matrix}$

Where {T} is the minimum portion of the background model. In oneexemplary implementation on both grayscale and RGB video inputs, forthree Gaussians (K=3), α may be set to 0.005, and T to 0.4.

Mixture of Gaussians methods and system may generate large areas offalse positive foreground when there are quick lighting changes. Someembodiments of the present invention address this issue by integratingthe texture information to the foreground mask for removing the falsepositive areas, as texture in the false positive foreground areas whichis caused by lighting changes should be similar to the texture in thebackground. The gradient value is less sensitive to lighting changes,enabling derivation of an accurate local texture difference measure.Thus, a texture similarity measure may be defined at pixel {X} between acurrent frame and a background image as:

$\begin{matrix}{{{S(X)} = \frac{\sum\limits_{u \in W_{x}}{2{{{g(u)}} \cdot {{g_{b}(u)}}}\cos\;\theta}}{\sum\limits_{u \in W_{x}}\left( {{{g(u)}}^{2} + {{g_{b}(u)}}^{2}} \right)}},} & (6)\end{matrix}$

where {W_(x)} denotes the {M by N} neighborhood centered at pixel {X},{g} and {g_(b)} is the gradient vector of the current frame and thebackground image respectively, and {θ} is the angle between the vectors.The gradient vector {g(X)=(g^(x)(X), g^(y)(X))} and the partialderivatives {g^(x)(X)} and {g^(y)(X)} are obtained by a Sobel operator.In the false positive foreground areas caused by quick lighting changes,there are no texture changes between the current frame and thebackground, therefore {S(X)≈1}. The foreground mask will be removed forthe areas with {S(X)≧T_(s)}. In some embodiments, the similaritythreshold is set as {T_(s)=0.7}.

Mixture of Gaussians methods and systems may use color information forshadow removal. Some embodiments of the present invention address thisissue with respect to grayscale images through use of intensityinformation instead of color information. For example, the normalizedcross-correlation of the intensities may be calculated at each pixel ofthe foreground region between the current frame and the backgroundimage. For pixel {X} in the {M by N} neighborhood, the normalizedcross-correlation may be calculated as:

$\begin{matrix}{{{{NCC}(X)} = \frac{\left( {{\sum\limits_{u \in W_{x}}{{I_{t}(u)} \cdot {I_{b}(u)}}} - {\frac{1}{MN}{\sum\limits_{u \in W_{x}}{{I_{t}(u)}{\sum\limits_{u \in W_{x}}{I_{b}(u)}}}}}} \right)}{\sqrt{\left( {{\sum\limits_{u \in W_{x}}{I_{t}^{2}(u)}} - {\frac{1}{MN}\left\lbrack {\sum\limits_{u \in W_{x}}{I_{t}(u)}} \right\rbrack}^{2}} \right)\left( {{\sum\limits_{u \in W_{x}}{I_{b}^{2}(u)}} - {\frac{1}{MN}\left\lbrack {\sum\limits_{u \in W_{x}}{I_{b}(u)}} \right\rbrack}^{2}} \right)}}},} & (7)\end{matrix}$

Where {W_(x)} denotes the {M by N} neighborhood centered at pixel {X},{I_(t)(u)} and {I_(b)(u)} is the intensity at pixel {u} of the currentframe and the background, respectively. The pixel {X} is shadow if{NCC(X)≧T_(s)} and {I_(t)(X≧T_(I)}, wherein the constraint{I_(t)(X≧T_(I)} may be added to avoid the detection of shadows in verydark areas. Otherwise, the pixel {X} is real foreground.

Referring again to FIG. 3, a low activity level indicated/detected at104 results in selection of a tracking-based analytic process/methodcomprising a color field tracker 124, an object classifier 122, a colorclassifier 126 and a tracking alert detector 128. The color fieldtracker 124 uses an appearance-based modeling to resolve complexstructures in a track lattice produced by bounding-box tracking. Moreparticularly, the foreground regions of each video frame are groupedinto connected components; in some embodiments, a size filter is used toremove small components. Each foreground component is described by abounding box and an image mask, which indicates those pixels in thebounding box that belong to the foreground, and wherein the set offoreground pixels may be designated {F}.

FIGS. 4 a and 4 b illustrate a bounding box distance measure accordingto the present invention. For each successive frame, a correspondenceprocess attempts to associate each foreground region with one existingtrack by constructing a distance matrix showing the distance betweeneach of the foreground regions and all the currently active tracks.Thus, the distance between bounding boxes A and B in FIG. 4 a is thelower of the distance from the centroid {C_(a)} of A to the closestpoint on B or from the centroid {C_(b)} of B to the closest point on A.If either centroid {C_(a)} or {C_(b)} lies within the other boundingbox, as shown in FIG. 4 b, then the distance is zero. In one aspect,using a bounding box distance as opposed to a Euclidean distance betweenthe centroids {C_(a)} and {C_(b)} avoids a large jump in the Euclideandistance when two bounding boxes or objects A and B merge or split. Atime distance between observations may also be added in, in one aspectto penalize tracks for which no evidence has been seen for some time.

The bounding box distance matrix may then be represented as binary data,resulting in a correspondence matrix associating tracks with foregroundregions and having rows corresponding to existing tracks and columnscorresponding to foreground regions in the current segmentation.Analysis of the correspondence matrix in one embodiment produces fourpossible results: an existing object, a new object, a merge detected anda split detected. More particularly, for well-separated moving objects,the correspondence matrix will have at most one non-zero element in eachrow or column, thus associating each track with one foreground regionand each foreground region with one track, respectively. Columns withall zero elements represent new objects in the scene which are notassociated with any track, and result in the creation of a new track.Rows with all zero elements represent tracks that are no longer visible(because they left the scene, or were generated because of artifacts ofthe background subtraction).

In the case of merging objects, two or more tracks may correspond to oneforeground region, i.e. a column in the correspondence matrix may havemore than one non-zero entry. When objects split, for example whenpeople in a group walk away from each other, a single track willcorrespond to multiple foreground regions, resulting in more than onenon-zero element in a row of the correspondence matrix. When a singletrack corresponds to more than one bounding box, all those boundingboxes are merged together, and processing proceeds. If two objectshitherto tracked as one should separate, the parts continue to betracked as one until they separate sufficiently that both bounding boxesdo not correspond to the track, and a new track is created.

Once a track is created, an appearance model of the object isinitialized. This appearance model is adapted every time the same objectis tracked into the next frame. On the detection of object merges, theappearance model is used to resolve the ambiguity.

For each track, the color field tracker 124 builds a red-green-blue(RGB) color appearance model {M_(RGB)(x)} representative of theappearance of each pixel {x} of an object, and an associated probabilitymask {P_(c)(x)} which represents the likelihood of the object beingobserved with respect to the pixel. For simplicity of notation, {x}represents the pixel coordinates, and which are assumed to be imagecoordinates, but in practice the appearance models model local regionsof the image only, normalized to the current centroid, which translatewith respect to the image coordinates. However, at any time an alignmentis known, allowing calculation of {P_(c)} and {M_(RGB)} for any point{x} in the image, and wherein {P_(c)(x)} is zero outside the modeledregion.

When a new track is created, a rectangular appearance model is createdwith the same size as the bounding box of the foreground region. Themodel is initialized by copying the pixels of the track's foregroundcomponent into the color model. The corresponding probabilities areinitialized to 0.4, and pixels which did not correspond to this trackare given zero initial probability.

On subsequent frames, the appearance model is updated by blending in thecurrent foreground region. The color model is updated by blending thecurrent image pixel with the color model for all foreground pixels, andall the probability mask values may be updated with the followingformulae (for α=λ=0.95):

$\begin{matrix}\begin{matrix}{{M_{RGB}\left( {x,t} \right)} = {{{{M_{RGB}\left( {x,{t - 1}} \right)}\alpha} + {\left( {1 - \alpha} \right){I(x)}\mspace{14mu}{if}\mspace{14mu} x}} \in}} \\{{P_{c}\left( {x,t} \right)} = {{{P_{c}\left( {x,{t - 1}} \right)}\lambda\mspace{14mu}{if}\mspace{14mu} x} \notin}} \\{= {{{{P_{c}\left( {x,{t - 1}} \right)}\lambda} + {\left( {1 - \lambda} \right)\mspace{14mu}{if}}} \in}}\end{matrix} & (8)\end{matrix}$

In this way, a continuously updated model of the appearance of thepixels in a foreground region may be maintained, together with theirobservation probabilities. Thresholds may be applied to the observationprobabilities, enabling treatment as a mask to find a boundary of theobject, and which also gives information about non-rigid variations inthe object, for instance retaining observation information about a wholeregion swept out by a pedestrian's legs.

The object classifier 122 labels objects in a scene through arules-based classifier. For example, objects may be classified by sizeand shape, or by type of object: single person, multiple people,vehicle, other, etc. Generally for each object, the object classifier122 finds an area, length of contour and length and orientation ofprincipal axes, and computes the “dispersedness” of the object, definedas the ratio of the perimeter squared to the area. Dispersedness hasbeen found useful in distinguishing two-dimensional (2D) image objectsof one or more people from those of individual vehicles. For each 2Dimage object, the object classifier 122 also determines which principalaxis is most nearly vertical and computes a ratio {r} of the more-nearlyhorizontal axis length to the more-nearly vertical axis length. FIG. 5provides an illustration of a rule-based classification derived from theratio {r}. For example, the ratio {r} may be used to distinguish aforeground region of a single person from one representing multiplepeople; a single person's image is typically significantly taller thanit is wide, while a multi-person blob grows in width with the number ofvisible people. In addition, temporal consistency may be used to improverobustness so that a cleanly tracked object, which is occasionallymisclassified, can use its classification history to improve results.

The color classifier 126 tracks objects of specified colors. The colorclassifier 126 may be built on top of the color field tracker 124, tothereby perform color quantization on a frame-by-frame level and providea specific color for each object tracked in a scene. In one embodiment,the color classifier 126 is a bi-conic color classifier that quantifiescolor information into colors by mapping RGB pixels for each video frameto a bi-conic Hue, Saturation, Lightness (HSL) color space defined by avertical Lightness axis value ranging from white (full brightness) toblack, angular Hue data, and radial color Saturation data. The HSL colorspace is quantified into colors by determining angular cutoffs betweencolors and lightness and saturation cutoffs, and then relabeling pixelsas either white or black depending on whether they lie outside a derivedlightness/saturation curve, or above or below a horizontal mid-plane inthe HSL space. In one embodiment, the color classifier 126 quantifiesthe color information into six colors (black, white, red, blue, green,and yellow) by providing four cutoffs between hues: yellow/green,green/blue, blue/red, and red/yellow. In one example, for an outdoorurban scene video input the cutoffs are 60°, 150°, 225°, and −15°. Thecolor classifier 126 further classified points above a horizontal planein the HSL space (i.e., for sufficient lightness) and having intensityand saturation outside a defined curve as white, and those below thehorizontal plane as black. Embodiments of the color classifier 126 mayfurther create an accumulated histogram of the quantized colors and thenselect a dominant color of the object, for example the color with thelargest number of votes in the histogram.

The tracking alert detector 128 generates alerts in response to objecttracking output by the color field tracker 124. In some embodiments, aregion of interest (ROI) is configured to represent a target region, andrules are specified to define region alert: for example, to trigger analert to an object initiated inside or outside of the ROI, an objectpassing through the ROI, an object entering the ROI from outside, or aspecified object ever being inside of the ROI. Location relativity mayalso be inferred by different parts of an object: for example, anobject's head or topmost point), a centroid, a foot part or lowestpoint, or a whole entirety of an object. Some embodiments may also applyone or more sizing thresholds to target objects in order to triggeralerts.

A high activity level indicated/detected at 104 results in the exampleof FIG. 3 in the selection of a non-tracking-based analyticprocess/method comprising a color BGS classifier 132 and a non-trackingalert detector 134, more particularly performing color retrieval withoutobject tracking. Some embodiments of the color BGS classifier 132utilize color segmentation inside the foreground objects detected usingbackground subtraction by applying a time interval (for example, twoseconds, three seconds, and other time intervals may be practiced) and asize threshold per color. For each foreground object, colors arequantified, for example through the methods and systems described abovewith respect to the bi-conic color classifier 126, and segmentation isperformed using connected component analysis for each color. Thus, foreach color detect in each time interval, if a connected component ofthat color is found which is bigger than a predefined size threshold, alargest component for that color in the time interval is stored as a keyframe for color retrieval.

With reference to FIG. 6, embodiments of the color BGS classifier 132may also define a virtual boundary line 120 (i.e., a virtual tripwire)in a video image from region of interest 119. The virtual boundary line120 is generally of arbitrary shape, which may be user-defined, and maybe placed in a digital video using computer-based video processingtechniques. Virtual boundary line 120 is monitored, statistics may becompiled, intrusions detected, events recorded, responses triggered,etc. More specifically the color BGS classifier 132 establishes a set ofoverlapping ground patch regions 125 along each side of the virtualboundary line 120. (It can be appreciated that each of the set of groundpatch regions 125 is capable of taking on any number of geometries(e.g., square, rectangle, etc.), and is not limited to the overlappingcircles depicted in FIG. 6).

The color BGS classifier 132 may thus process video data in real-time,identifying attributes of objects detected in the region of interest119. Objects can be detected using a number of approaches, including,but not limited to: background modeling, object detection and tracking,spatial intensity field gradient analysis, diamond search block-based(DSBB) gradient descent motion estimation, or any other method fordetecting and identifying objects captured by a sensor device. In anexemplary embodiment, the color BGS classifier 132 analyzes each groundpatch region 125 to identify foreground objects within, and thencomputes current appearance features of the identified foregroundobjects. Specifically, a ground patch history model is updated withattributes extracted from each of set of ground patch regions 125. In anexemplary embodiment, extraction relates each attribute to a groundpatch history model according to various attributes, including, but notlimited to, appearance, color, texture, gradients, edge detection,motion characteristics, shape, spatial location, etc. Data associatedwith each of the extracted attributes is dynamically mapped intogroups/models for each ground patch region, along with additionalmetadata that captures a more detailed description of the extractedattribute and/or objects. For example, one ground patch history modelmay comprise information about each ground patch region, including, butnot limited to: ground patch region center location, ground patch regionradius, timestamp, frame number, a list of history patch models (e.g.,color histograms, appearance features, etc.), a list of neighboringground patch regions in spatial proximity and/or on an opposite side ofthe virtual boundary line, or a patch motion vector indicating thegeneral direction of motion for each ground patch region.

The ground patch history model is continuously updated andcross-referenced against attributes from previously received sensor data(i.e., video input) to determine if each ground patch region'sappearance has changed. To accomplish this, the color BGS classifier 132analyzes the updated ground patch history model to detect whether anobject captured in at least one of a set of ground patch regions 125 iscrossing virtual boundary line 120 in the video image. Specifically, thecolor BGS classifier 132 may analyze appearance features within theground patch history model for each of a set of ground patch regions 125and determine if a pair of similar ground patch regions is present amongthe set of ground patch regions 125 based on the appearance featureswithin ground patch history model for each of set of ground patchregions 125.

The color BGS classifier 132 further determines locations of pairs ofsimilar ground patch regions. For example, appearance similarities arecompared between a specific patch being analyzed and neighboring groundpatch regions on the other side of virtual boundary line 120. Directionsof motion for each pair are compared, in the case that each of the pairis located on a different side of virtual boundary line 120 in the videoregion of interest 119. In some embodiments, pairs of similar groundpatch regions are matched by imposing an attribute similarityconstraint, the direction of the object movements in the pair thenestimated using techniques such as optical flow estimation, affinetransformation, smallest squared difference (SSD), etc. If the matchedpaired ground patch regions are due to the same object, their movingdirection should be consistent (i.e., both patches have movement vectorspointing to the same side of the virtual boundary line), but if the pairmatching is caused by different objects with similar appearances, thematched pair's moving directions will typically be different. Thus, byimposing this motion direction agreement constraint, false positivescaused by matching different objects are reduced. To determine if anobject is moving in a required direction, its motion direction may becompared with a virtual boundary line crossing direction 130 (e.g.,inside (+) to outside (−), or vice versa).

Alerts are generated by the non-tracking alert detector 134 if a matchis determined between the virtual boundary line crossing direction 130and an object motion direction, the object detected as crossing thevirtual boundary line 120 in the video region of interest 119, if thedirection of motion for each of the pair of similar ground patch regionsis substantially the same. Further, it will be noted that motiondetection alerts by the non-tracking alert detector 134 may be triggeredwhen the target region-of-interest (ROI) 119 possesses a sufficientamount of motion energy that lasts within a desired temporal interval,which may be selected or revised as needed, for example, one second, twoseconds, five minutes, etc. Applications of this feature includeloitering detection, ROI occupancy estimation, and object accessdetection. In urban scenes, the non-tracking alert detector 134 mayprovide simplified abandoned object alert, for example where parkedvehicles may be detected by specifying an ROI 119 around a parking area.In contrast to the tracking alert detector 128 of the low-levelanalytics mode, motion detection alerts by the non-tracking alertdetector 134 may consider the global motion energy of the ROI withoutdistinction of individual objects.

The index writer 136 receives input from the non-tracking alert detector134 and the tracking alert detector 128, and further receives data fromthe color classifier 126 and the object classifier 122 with respect tothe tracking alert detector 128 data. In addition to providing real-timealerts and indexing thereof, the index writer 136 also enables datasearching based on attributes extracted from the low and high levelanalytics, for example object type (person, vehicle), color, size,speed, human body parts, and many others. In some embodiments, theseattributes are constantly ingested as XML metadata into a DB2 databaseas new events are detected. In one aspect, the index writer 136 enablescomposite searching by combining different visual attributes or evennon-visual data captured from multiple data sources.

Referring now to FIG. 7, an exemplary computerized implementation of anembodiment of the present invention includes computer or otherprogrammable devices 304 in communication with devices 336 (for example,a video camera or video server) that analyzes video data fordetermination of object movement according to the present invention, forexample in response to computer readable code 302 in a file residing ina memory 316 or a storage system 332 through a computer networkinfrastructure 308. The implementation is intended to demonstrate, amongother things, that the present invention could be implemented within anetwork environment (e.g., the Internet, a wide area network (WAN), alocal area network (LAN) or a virtual private network (VPN), etc.)Communication throughout the network 308 can occur via any combinationof various types of communication links: for example, communicationlinks can comprise addressable connections that may utilize anycombination of wired and/or wireless transmission methods.

Where communications occur via the Internet, connectivity could beprovided by conventional TCP/IP sockets-based protocol, and an Internetservice provider could be used to establish connectivity to theInternet. Still yet, the network infrastructure 308 is intended todemonstrate that an application of an embodiment of the invention can bedeployed, managed, serviced, etc. by a service provider who offers toimplement, deploy, and/or perform the functions of the present inventionfor others.

The computer 304 comprises various components, some of which areillustrated within the computer 304. More particularly, as shown, thecomputer 304 includes a processing unit (CPU) 312 in communication withone or more external I/O devices/resources 328 and storage systems 332.In general, the processing unit 312 may execute computer program code,such as the code to implement one or more of the process stepsillustrated in FIG. 1, which is stored in the memory 316 and/or thestorage system 332.

The network infrastructure 308 is only illustrative of various types ofcomputer infrastructures for implementing the invention. For example, inone embodiment, computer infrastructure 308 comprises two or morecomputing devices (e.g., a server cluster) that communicate over anetwork. Moreover, the computer 304 is only representative of variouspossible computer systems that can include numerous combinations ofhardware. To this extent, in other embodiments, the computer 304 cancomprise any specific purpose computing article of manufacturecomprising hardware and/or computer program code for performing specificfunctions, any computing article of manufacture that comprises acombination of specific purpose and general purpose hardware/software,or the like. In each case, the program code and hardware can be createdusing standard programming and engineering techniques, respectively.

Moreover, the processing unit 312 may comprise a single processing unit,or be distributed across one or more processing units in one or morelocations, e.g., on a client and server. Similarly, the memory 316and/or the storage system 332 can comprise any combination of varioustypes of data storage and/or transmission media that reside at one ormore physical locations. Further, I/O interfaces 328 can comprise anysystem for exchanging information with one or more of an external serverand or client (not shown). Still further, it is understood that one ormore additional components (e.g., system software, math co-processingunit, etc.) not shown can be included in the computer 304 or server orclient.

One embodiment performs process steps of the invention on asubscription, advertising, and/or fee basis. That is, a service providercould offer to provide automated analysis of video data fordetermination of object movement. In this case, the service provider cancreate, maintain, and support, etc., a computer infrastructure, such asthe network computer infrastructure 308 that performs the process stepsof the invention for one or more customers. In return, the serviceprovider can receive payment from the customer(s) under a subscriptionand/or fee agreement and/or the service provider can receive paymentfrom the sale of advertising content to one or more third parties.

In still another embodiment, the invention provides acomputer-implemented method for executing one or more of the processes,systems and articles for automated analysis of video data fordetermination of object movement described above. In this case, acomputer infrastructure, such as the computer infrastructure 308, can beprovided and one or more systems for performing the process steps of theinvention can be obtained (e.g., created, purchased, used, modified,etc.) and deployed to the computer infrastructure. To this extent, thedeployment of a system can comprise one or more of: (1) installingprogram code on a computing device, such as the computers/devices304/336, from a computer-readable medium; (2) adding one or morecomputing devices to the computer infrastructure; and (3) incorporatingand/or modifying one or more existing systems of the computerinfrastructure to enable the computer infrastructure to perform theprocess steps of the invention.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof. As used herein, it is understood thatthe terms “program code” and “computer program code” are synonymous andmean any expression, in any language, code or notation, of a set ofinstructions intended to cause a computing device having an informationprocessing capability to perform a particular function either directlyor after either or both of the following: (a) conversion to anotherlanguage, code or notation; and/or (b) reproduction in a differentmaterial form. To this extent, program code can be embodied as one ormore of: an application/software program, component software/a libraryof functions, an operating system, a basic I/O system/driver for aparticular computing and/or I/O device, and the like.

Certain examples and elements described in the present specification,including in the claims and as illustrated in the Figures, may bedistinguished or otherwise identified from others by unique adjectives(e.g. a “first” element distinguished from another “second” or “third”of a plurality of elements, a “primary” distinguished from a“secondary,” one or “another” item, etc.) Such identifying adjectivesare generally used to reduce confusion or uncertainty, and are not to beconstrued to limit the claims to any specific illustrated element orembodiment, or to imply any precedence, ordering or ranking of any claimelements, limitations or process steps.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present invention has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the invention. Theembodiment was chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

What is claimed is:
 1. A method for multi-mode video event indexing, themethod comprising: applying background subtraction via a programmablehardware device to a video stream input to detect foreground objectactivity as a difference between current frame image information andstatic information of a contextual background of video input images fromthe video stream input; determining a level of the detected foregroundobject activity with respect to the video input images from the videostream input; selecting from a plurality of video analytics modes andapplying an object tracking-based analytic mode via a hardware device tothe detected foreground object activity of the video input images totrack a foreground object, in response to the determined level ofdetected quality of the detected foreground object activity meeting athreshold level of quality; selecting from the plurality of videoanalytics modes and applying a non-object tracking-based analytic modeto the detected foreground object activity of the video input images viaa hardware device to determine object movement from extracted foregroundobject appearance attributes without tracking the foreground object, inresponse to the determined level of detected quality of the detectedforeground object activity not meeting the threshold level of quality;and writing results of the tracking the foreground object and the objectmovement determination from the extracted foreground object appearanceattributes to an index.
 2. The method of claim 1, wherein thedetermining the level of the detected foreground object activitycomprises: determining a density of the foreground relative to anentirety of the image; and wherein the threshold level of activity is adensity value.
 3. The method of claim 2, wherein the selecting theobject tracking-based mode further comprises: determining if a firstdensity of the foreground relative to the entirety of the image isgreater than or equal to the threshold density value at a first time;determining if a second density of the foreground relative to theentirety of the image is less than a minimum value that is less than thethreshold density value at a second time subsequent to the first time;determining an elapsed time between the first and the second times;comparing the elapsed time to a stability time period; and selecting theobject tracking-based mode if the elapsed time is less than thestability time period.
 4. The method of claim 2, wherein the selectingthe object tracking-based mode further comprises: determining a timeduration that the density of the foreground is greater than or equal tothe threshold level of activity; comparing the time duration to aspecified time period; selecting the object tracking-based mode if thetime duration is less than the specified time period.
 5. The method ofclaim 4, wherein the density of the foreground is an average of aplurality of individual densities over the specified time period.
 6. Themethod of claim 5, further comprising: determining the average of theplurality of individual densities over the specified time period byweighting a more recent density higher than an older density.
 7. Acomputer system for multi-mode video event indexing, the computer systemcomprising: a processing unit, computer-readable memory and acomputer-readable storage system; fourth program instructions to applybackground subtraction to a video stream input to detect foregroundobject activity as a difference between current frame image informationand static information of a contextual background of video input imagesfrom the video stream input; first program instructions to determine thequality of object distinctiveness by determining a level of the detectedforeground object activity with respect to the video input images fromthe video stream input; second program instructions to select from aplurality of video analytics modes and apply an object tracking-basedmode to the detected foreground object activity of the video input totrack a foreground object, in response to the determined level ofdetected quality of object distinctiveness meeting a threshold level ofquality; third program instructions to select from the plurality ofvideo analytics modes and apply a non-object tracking-based to thedetected foreground object activity of the video input images todetermine object movement from extracted foreground object appearanceattributes without tracking the foreground object, in response to thedetermined level of detected quality of the detected foreground objectactivity not meeting the threshold level of quality; and fifth programinstructions to write results of the tracking the foreground object andthe object movement determination from the extracted foreground objectappearance attributes to an index; and wherein the first, second, third,fourth and fifth program instructions are stored on thecomputer-readable storage system for execution by the processing unitvia the computer-readable memory.
 8. The computer system of claim 7,wherein the first program instructions are further to determine thelevel of the detected foreground object activity as a density of theforeground relative to an entirety of the image; and wherein thethreshold level of activity is a density value.
 9. The computer systemof claim 8, wherein the second program instructions are to select andapply the object tracking-based mode by: determining a time durationthat the density of the foreground is greater than or equal to thethreshold level of activity; comparing the time duration to a specifiedtime period; and selecting the object tracking-based mode if the timeduration is less than the specified time period.
 10. The computer systemof claim 8, wherein the second program instructions are to select andapply the object tracking-based mode by: determining if a first densityof the foreground relative to the entirety of the image is greater thanor equal to the threshold density value at a first time; determining ifa second density of the foreground relative to the entirety of the imageis less than a minimum value that is less than the threshold densityvalue at a second time subsequent to the first time; determining anelapsed time between the first and the second times; comparing theelapsed time to a stability time period; and selecting the objecttracking-based mode in response to determining that the elapsed time isless than the stability time period.
 11. The computer system of claim10, wherein the second program instructions are further to determine thedensity of the foreground as an average of a plurality of individualdensities over the specified time period.
 12. The computer system ofclaim 11, wherein the second program instructions are further todetermine the average of the plurality of individual densities over thespecified time period by weighting a more recent density higher than anolder density.
 13. A computer program product for multi-mode video eventindexing, the computer program product comprising: a computer-readabletangible storage device having computer-readable program code embodiedtherewith, the computer-readable program code comprising instructionsthat, when executed by a computer processing unit, cause the computerprocessing unit to: apply background subtraction to a video stream inputto detect foreground object activity as a difference between currentframe image information and static information of a contextualbackground of video input images from the video stream input; determinea level of the detected foreground object activity with respect to thevideo input images from the video stream input; select from a pluralityof video analytics modes and apply an object tracking-based analyticmode to the detected foreground object activity of the video inputimages to track a foreground object, in response to the determined levelof detected quality of the detected foreground object activity meeting athreshold level of quality; select from the plurality of video analyticsmodes and apply a non-object tracking-based analytic mode to thedetected foreground object activity of the video input images todetermine object movement from extracted foreground object appearanceattributes without tracking the foreground object, in response to thedetermined level of detected quality of the detected foreground objectactivity not meeting the threshold level of quality; and write resultsof the tracking of the foreground object and the object movementdetermination from the extracted foreground object appearance attributesto an index.
 14. The computer program product of claim 13, wherein thecomputer-readable program code instructions, when executed by thecomputer processing unit, further cause the computer processing unit todetermine the level of the detected foreground object activity bydetermining a density of the foreground relative to an entirety of theimage; and wherein the threshold level of activity is a density value.15. The computer program product of claim 14, wherein thecomputer-readable program code instructions, when executed by thecomputer processing unit, further cause the computer processing unit toselect the object tracking-based mode in response to: determining if afirst density of the foreground relative to the entirety of the image isgreater than or equal to the threshold density value at a first time;determining if a second density of the foreground relative to theentirety of the image is less than a minimum value that is less than thethreshold density value at a second time subsequent to the first time;determining an elapsed time between the first and the second times;comparing the elapsed time to a stability time period; and selecting theobject tracking-based mode in response to determining that the elapsedtime is less than the stability time period.
 16. The computer programproduct of claim 15, wherein the computer-readable program codeinstructions, when executed by the computer processing unit, furthercause the computer processing unit to select the object tracking-basedmode by: determining a time duration that the density of the foregroundis greater than or equal to the threshold level of activity; comparingthe time duration to a specified time period; selecting the objecttracking-based mode in response to determining that the time duration isless than the specified time period.
 17. The computer program product ofclaim 16, wherein the computer-readable program code instructions, whenexecuted by the computer processing unit, further cause the computerprocessing unit to determine the density of the foreground as an averageof a plurality of individual densities over the specified time period.18. The computer program product of claim 17, wherein thecomputer-readable program code instructions, when executed by thecomputer processing unit, further cause the computer processing unit todetermine the average of the plurality of individual densities over thespecified time period by weighting a more recent density higher than anolder density.
 19. A method of providing a service for multi-mode videoevent indexing, the method comprising: integrating computer-readableprogram code into a computer system comprising a processing unit, acomputer-readable memory and a computer-readable tangible storagedevice; wherein the computer-readable program code is embodied on thecomputer-readable tangible storage device and comprises instructionsthat, when executed by the processing unit via the computer-readablememory, cause the processing unit to: apply background subtraction to avideo stream input to detect foreground object activity as a differencebetween current frame image information and static information of acontextual background of video input images from the video stream input;determine a level of the detected foreground object activity withrespect to the video input images from the video stream input; selectfrom a plurality of video analytics modes and apply an objecttracking-based analytic mode to the detected foreground object activityof the video input images to track a foreground object, in response tothe determined level of detected quality of the detected foregroundobject activity meeting a threshold level of quality; select from theplurality of video analytics modes and apply a non-object tracking-basedanalytic mode to the detected foreground object activity of the videoinput images to determine object movement from extracted foregroundobject appearance attributes without tracking the foreground object, inresponse to the determined level of detected quality of the detectedforeground object activity not meeting the threshold level of quality;and write results of the tracking of the foreground object and theobject movement determination from the extracted foreground objectappearance attributes to an index.
 20. The method of claim 19, whereinthe computer-readable program code instructions, when executed by thecomputer processing unit, further cause the computer processing unit todetermine the level of the detected foreground object activity bydetermining a density of the foreground relative to an entirety of theimage; and wherein the threshold level of activity is a density value.21. The method of claim 20, wherein the computer-readable program codeinstructions, when executed by the computer processing unit, furthercause the computer processing unit to select the object tracking-basedmode in response to: determining if a first density of the foregroundrelative to the entirety of the image is greater than or equal to thethreshold density value at a first time; determining if a second densityof the foreground relative to the entirety of the image is less than aminimum value that is less than the threshold density value at a secondtime subsequent to the first time; determining an elapsed time betweenthe first and the second times; comparing the elapsed time to astability time period; and selecting the object tracking-based mode inresponse to determining that the elapsed time is less than the stabilitytime period.
 22. The method of claim 21, wherein the computer-readableprogram code instructions, when executed by the computer processingunit, further cause the computer processing unit to select the objecttracking-based mode by: determining a time duration that the density ofthe foreground is greater than or equal to the threshold level ofactivity; comparing the time duration to a specified time period;selecting the object tracking-based mode in response to determining thatthe time duration is less than the specified time period.
 23. The methodof claim 22, wherein the computer-readable program code instructions,when executed by the computer processing unit, further cause thecomputer processing unit to determine the density of the foreground asan average of a plurality of individual densities over the specifiedtime period.
 24. The method of claim 23, wherein the computer-readableprogram code instructions, when executed by the computer processingunit, further cause the computer processing unit to determine theaverage of the plurality of individual densities over the specified timeperiod by weighting a more recent density higher than an older density.